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Abstract 


Information  Operations  (InfoOps)  can  be  used  by  or  affect  all  kinds  of  actors  in  a 
multinational,  civil-military  operation.  To  explore  issues  related  to  command  and  control 
(C2)  and  information  operations,  a  simulation  environment  was  developed  in  which  a 
series  of  experiments  was  performed.  The  simulator  is  called  the  Command  and  Control 
Warfare  Demonstrator  (C2WD).  In  the  C2WD,  C2  warfare  in  terms  of  electronic  warfare 
(EW)  and  computer  network  operations  (CNO)  can  be  used  to  facilitate  or  reduce  certain 
aspects  of  a  staffs  C2  ability.  During  experiments,  military  InfoOps  experts  formed  a 
team  responsible  for  C2W  resources  in  a  Active  peace  keeping  mission.  The  scenario  was 
developed  as  being  comprehensive,  involving  both  civil  and  military  aspects.  Data 
collection  was  based  on  a  hypothesis  of  ten  prerequisites  for  C2  and  included 
performance  measures,  self  estimations,  observations,  system  logs,  video  and  screen 
recordings.  A  synchronized  replay  of  all  data  sources  was  performed  in  order  to  compare 
the  results  from  the  different  sources.  The  main  findings  are  that  the  ten  prerequisites  for 
C2  seem  to  explain  C2  ability.  Based  on  these  findings,  a  LISREL  model  was  developed, 
where  four  factors  with  causal  connections  were  created,  visualizing  the  relations 
between  these  prerequisites. 


Introduction 

Military  Command  and  Control  may  be  defined  as  “the  Organisation,  Process, 

Procedures  and  Systems  necessary  to  allow  timely  political  and  military  decision  making 
and  to  enable  military  commanders  to  direct  and  control  military  forces”  (NATO,  2004). 
The  ability  to  make  these  decisions  are  dependent  on  a  number  of  factors  such  as  access 
to  correct  and  relevant  information,  understanding  of  the  mission  intent  understanding  of 
the  situation  in  which  the  mission  takes  place  and  an  adequate  information  flow  between 
subordinates  and  superiors.  This  paper  aims  to  describe  the  methodology  and  results  of 
studying  C2  ability  within  the  setting  of  a  C2  Warfare  Demonstrator  (C2WD). 

During  the  development  process  of  the  C2WD,  technology  was  developed 
simultaneously  with  methods  for  assessing  command  and  control  ability.  A  hypothesis  of 
prerequisites  for  C2  was  developed,  based  on  current  research  as  well  as  doctrines  and 
expertise  knowledge  regarding  command  and  control  processes  in  the  Armed  Forces 
(Hammervik  et.  al,  2007;  Hammervik  et.  al,  2009).  Earlier  experiments  in  the 
demonstrator  environment  resulted  in  revision  of  the  original  hypothesis  in  that  new 
prerequisites  were  added  and  refonnulated,  but  it  has  never  been  found  that  any  of  the 
prerequisites  would  be  redundant  or  unnecessary.  The  current  experiments  are  based  on 
the  revised  hypothesis,  which  contained  ten  prerequisites  (Figure  1). 
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Figure  1.  Hypothesis  of  the  10  prerequisites  for  Command  and  Control  Ability.  C2  warfare  (in  terms 
of  CNO  and  EW)  may  affect  C2  ability,  which  is  constituted  of  10  prerequisites.  In  the  demonstrator 
environment,  this  is  studied  qualitatively  and  quantitatively,  using  both  objective  and  subjective 
measures. 

There  is  support  in  research  and  literature  for  that  the  terms  that  are  called  prerequisites 
are  central  aspects  constituting  C2  ability.  Decision  making  is  a  huge  research  area  in 
which  Klein  (Klein  et.  al,  1993)  and  Rasmussen  (Rasmussen,  Brehmer  &  Leplat,  1991) 
are  two  central  contributors.  The  area  of  situational  awareness  is  complex,  yet  well 
recognized  tenn  within  human  factors  (Endsley,  Bolte  &  Jones,  2003),  which  has  been 
studied  in  different  areas  such  as  military  systems  (Matthews  et.  al,  2000)  teamwork 
(Bolstad  &  Endsley,  2000),  decision  making  (Endsley,  2004)  and  electronic  warfare 
(Riley,  Kaber  &  Draper,  2004).  Feedback  is  shown  to  affect  performance  both  positively 
and  negatively,  depending  on  type  of  feedback  given  as  well  as  delays  in  the  feedback 
(Lim,  O’Connor  &  Remus,  2005;  Atkins,  Wood  &  Rutgers,  2002;  Brehmer,  1989,  1992). 
Collaboration  and  teamwork  is  another  central  area  (Janis,  Cannon-Bowers  &  Salas, 
1998;  Beaubien,  Baker  &  Holtzman,  2003;  Brannick,  Salas  &  Prince,  1998)  which  also 
has  been  studied  nationally  in  a  Swedish  military  context  (Hoglund,  Berggren  & 
Nahlinder,  2009;  Svensson  &  Andersson,  2006;  Cheah  et.  al,  2005).  Also  trust  has  been 
identified  both  nationally  and  internationally  as  a  central  aspect  in  command  and  control 
(Andersson,  Malm  &  Thuren,  2003;  Lee  &  See,  2004). 

The  Swedish  Defence  Research  Agency  was  tasked  by  the  Swedish  Armed  Forces  to 
develop  a  Command  and  Control  Warfare  Demonstrator  with  the  purpose  of  being  a 
platform  for  studying  C2  ability  in  a  C2  warfare  setting,  where  C2  warfare  mainly 
focused  on  electronic  warfare  (EW)  and  computer  network  operations  (CNO).  In  the 
demonstrator  environment,  C2  warfare  can  be  used  to  impair  the  C2  ability  of  an 
opponent  or  to  improve  one’s  own  C2  ability  (Tyden,  et.  al,  2009).  This  can  be  done  by  a 
number  of  means  including  to  disturb  the  opponent’s  lines  of  communication,  plant  false 
information  into  the  opponent’s  information  systems  and  explore  infonnation  about  the 
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intentions  or  capabilities  of  an  opponent.  The  demonstrator  has  been  developed  using  an 
evolutionary  approach  where  experiments  were  used  for  validation  of  the  current  version 
of  the  technical  platform  as  well  as  collecting  requirements  for  further  development  of 
technology,  scenarios  and  methods  for  assessing  command  and  control  ability. 
Methodological  issues  related  to  the  C2WD  have  addressed  the  key  question  of  how  the 
effects  of  C2  warfare  on  C2  ability  can  be  assessed. 

This  paper  describes  two  experiments  in  which  C2  ability  has  been  studied  within  a 
demonstrator  setting,  and  also  reports  results  of  the  analysis  of  C2  ability.  During  the 
experiments,  a  simulated  environment  in  fonn  of  the  C2WD  was  used,  focusing  on 
Electronic  Warfare  (EW)  and  Computer  Network  Operations  (CNO)  for  exploitation, 
countermeasures  or  counter-countermeasures. 


Method 


Participants 

Two  similar  experiments  were  conducted.  In  each  experiment  a  military  staff  was 
formed,  consisting  of  three  persons  in  the  first  experiment  and  four  persons  in  the  second 
experiment.  All  staff  members  were  employed  by  the  Swedish  Armed  Forces  and 
working  with  information  operation  related  issues  (EW,  CNO,  psychological  operations, 
PSYOPS).  Five  were  military  officers  and  two  were  civilian  employees.  The  composed 
staffs  were  estimated  as  comparable  in  tenns  of  competence  and  familiarity,  in  that 
participants  in  each  staff  were  familiar  with  each  other. 

Procedure 

Each  staff  received  a  full  day  of  briefings  and  training  in  the  demonstrator  environment 
one  day  before  their  respective  experiment  started.  Each  experiment  (simulation)  lasted 
for  two  days  with  break  during  the  night.  During  the  experiments,  two  assistants  from  the 
C2WD  development  team  were  responsible  for  making  inputs  into  the  technical  platfonn, 
management  of  web  pages  and  e-mail,  on  orders  from  the  staff.  Operational  picture,  web 
pages  and  e-mail  interface  were  projected  in  front  of  the  staff.  A  higher  chief  in  command 
(HIC)  orally  communicated  the  mission  order  and  situation  briefing  before  the  simulation 
started.  During  the  simulation,  the  staff  and  HIC  communicated  via  e-mail.  The  staff 
reported  their  activities  and  asked  questions  to  HIC,  and  they  received  orders  and 
information  from  the  HIC.  A  game  control  team  simulated  the  opposing  forces  and 
coordinating  units  in  the  scenario.  HIC  coordinated  all  injects  towards  the  staff.  Staff 
members,  staff  assistants,  game  control  team,  HIC  and  observers  were  all  located  in  the 
same  room  during  the  experiment  (Figure  2). 
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Figure  2.  Arrangement  of  personnel  and  equipment  during  the  experiments. 

Scenario  and  simulated  environment 

The  scenario  took  place  in  a  Active  country  in  which  an  international  task  force  recently 
had  deployed.  The  participants  played  staff  of  a  subunit  of  the  taskforce,  responsible  for 
the  EW  and  CNO  resources,  such  as  units  with  the  capability  of  performing  signal 
intelligence,  scanning  computer  networks,  and  active  means  such  as  radio  and  radar 
jamming.  The  task  was  to  support  the  evacuation  of  three  NGO:s  (non-govemmental 
organizations)  and  key  government  personnel.  The  scenario  included  two  phases,  the 
intelligence  phase  and  the  evacuation  phase.  The  task  for  the  first  simulation  day  (i.e. 
intelligence  phase)  was  to  map  activities  in  the  operational  area  that  could  pose  a  threat  to 
the  task  force  or  the  evacuation  of  NGO/govemment  personnel.  Based  on  the  information 
obtained  during  the  intelligence  phase,  the  staff  recommended  evacuation  routes  and 
suggested  necessary  actions  to  ensure  the  evacuations.  The  task  for  the  second  day  (i.e. 
evacuation  phase)  was  to  monitor  the  evacuation,  report  threats  and  perform  necessary 
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activities  in  order  to  secure  the  evacuation  (e.g.  jamming).  Threats  towards  the  task  force 
included  both  conventional  military  units  and  criminal,  irregular  actors,  which  also 
possessed  EW-  and  CNO  resources.  Mapping  of  these  criminal  actors,  their  resources  and 
their  social  network  were  essential  tasks  for  the  staff.  The  staff  obtained  information 
mainly  by  using  signal  intelligence  (i.e.  radio,  radar,  and  mobile  telephone 
communication),  monitoring  social  media  and  web  pages,  and  intelligence  reports  from 
HIC. 

The  version  of  the  C2WD  used  in  the  experiment  included  a  wide  range  of  functionality 
to  perfonn  simulation  in  a  C2  Warfare  setting.  Particularly,  the  EW  models  were 
advanced,  which  meant  that  many  real  world  physical  effects  could  be  simulated.  During 
the  experiments,  the  staffs  were  able  to  acquire  information  using  active  and  passive 
sensors  such  as  IRST,  radar  and  radio  intelligence,  perfonn  electronic  counter  measures 
(against  radar,  radio  and  mobile  telephones)  and  use  different  kinds  of  CNO 
functionality,  including  passive  and  active  network  scan,  DOS  (Denial  of  Service)  attacks 
and  planting  spyware.  The  different  actors  in  the  scenario  were  prc-defincd  in  terms  of 
platforms  for  air,  land  and  sea  equipped  with  sensors,  countenneasures  and 
communication  equipment  with  certain  ranges.  The  three-dimensional  terrain  model 
affected  sensor  perfonnance  and  radio  range  in  a  realistic  way.  The  staff  also  used  the 
demonstrator  environment  for  electronically  sending  and  receiving  messages  (orders, 
reports)  and  monitor  Active  web  pages  that  were  part  of  the  scenario.  The  demonstrator 
environment  allowed  information  to  be  shown  on  different  types  of  views  that  were 
projected  simultaneously  on  different  screens.  Own  and  detected  units  were  shown  on  a 
geographical  map,  but  there  were  also  logical  network  and  communication  views  used  for 
instance  for  the  social  network  analysis,  as  well  as  web  and  e-mail  interfaces. 

Visualising  the  effects  of  CNO  and  EW  was  critical,  since  the  staff  needed  to  be  able  to 
detect  exposure  to  hostile  measures  in  order  to  be  able  to  calculate  risks  and  take  effects 
in  order  to  solve  the  task  of  evacuation  of  the  NGO/govemment  personnel.  Radio 
communication  jamming  could  be  detected  by  the  staff  in  that  they  were  unable  to 
communicate  with  units  in  a  certain  frequency  spectrum.  Certain  units  were  equipped 
with  a  Blue  Force  Tracking  system,  which  meant  that  they  automatically  sent  reports  of 
their  position.  If  a  subordinate  unit  that  was  supposed  to  be  under  transport  suddenly 
seemed  to  stop  in  the  operational  picture,  it  could  be  a  sign  of  jamming  connected  to 
GPS.  Radar  jamming  was  simulated  as  DRFM  (digital  radio  frequent  memory),  which 
meant  that  several  false  target  appeared  in  the  operational  picture.  For  detection  of 
intrusion  in  computer  systems,  the  staff  could  activate  IDS  (intrusion  detection  systems). 
However,  especially  for  CNO,  attacks  are  not  always  easily  detected.  For  those  cases, 
secondary  effects  were  used,  so  that  the  staff  would  be  able  to  note  that  they  had  been 
attacked.  Such  effects  included  that  internal  system  information  about  the  international 
force  was  published  on  the  irregulars’  home  page,  e-mails  with  strange  text  (although 
with  a  trusted  sender)  appeared  in  the  mail  box  or  information  on  a  public  news  web  page 
was  replaced  by  information  promoting  the  irregular  force.  The  staff  also  received 
intelligence  reports  to  guide  them  towards  a  certain  threat  or  area.  In  sum,  to  be  able  to 
detect  exposure  to  attacks,  the  staff  needed  not  only  to  activate  automatic  warning  system 
but  also  actively  scan  their  systems  and  the  operational  picture  for  indications,  question 
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the  information  seen  and  use  information  from  several  sources.  A  subordinate  unit  not 
responding  to  radio  calls  could  be  a  sign  of  jamming,  but  it  could  also  be  deployed  out  of 
range. 

Data  collection 

Data  collection  consisted  of  both  subjective  and  objective  data  in  form  of  observations, 
staff  members’  self  estimations  and  questionnaires  before,  during  and  after  the 
experiment,  measurement  of  performance  using  an  objective  performance  measure, 
system  logs,  video  and  audio  recordings,  and  recordings  of  all  screens  used  by  the  staff, 
HIC  and  game  control.  During  experiments,  two  observers  were  responsible  for 
command  and  control  aspects  and  two  observers  were  responsible  for  observing  the 
communication.  Observers  used  pre-defincd  schemes  created  in  the  observation  software 
NBOT  (Network-Based  Observation  Tool),  developed  by  the  Swedish  Defence  Research 
Agency  (Thorstensson,  2008). 

An  individual  background  survey  was  completed  by  the  staff  members  in  order  to  obtain 
information  about  their  knowledge  and  experience  within  EW,  CNO  and  staff  work.  A 
survey  was  also  completed  by  the  staff  members  after  each  day,  containing  questions 
related  to  the  prerequisites  for  C2  and  evaluation  of  the  experiment.  Three  to  four  times 
each  simulation  day,  observers  and  staff  members  completed  a  survey  of  the  prerequisites 
for  C2,  estimating  to  which  extent  the  prerequisites  for  C2  were  fulfilled  on  a  scale  from 
one  to  seven.  During  the  first  experiment,  the  survey  of  prerequisites  was  completed 
seven  times  and  during  the  second  eight  times.  Although  there  were  ten  prerequisites  for 
C2,  the  survey  contained  13  items,  since  the  prerequisite  Trust  was  divided  into  trust  for 
staff  assistant,  other  staff  members  and  technology  used.  Furthermore,  C2  as  a  whole  was 
rated. 

During  the  simulation,  one  of  the  staff  members  was  also  responsible  for  reporting 
experienced  CNO,  radar  jamming  and  communication  jamming  towards  the  force.  This 
survey  was  taken  every  four  minutes.  Estimations  were  graded  as  1)  No  jamming/CNO, 
2)  jamming/CNO  with  minor  impact  on  staffs  C2  ability,  and  3)  jamming/CNO  with 
major  impact  on  staffs  C2  ability. 

An  objective  performance  measure  was  developed  based  on  the  simulation  manuscript 
(describing  all  events  and  injects  towards  the  staff),  assigning  scores  based  on  how  the 
staff  reacted  to  the  events.  The  staff  assigned  scores  for  detecting,  identifying  and 
reporting  events.  Scoring  was  conducted  afterwards  using  system  logs,  e-mail  logs  and 
synchronized  replays  of  the  simulation.  For  replaying  the  simulations  and  the  chain  of 
events  that  had  occurred  during  the  exercise,  an  in-house  developed  multi-media  tool  and 
methods  for  synchronized  reconstruction  called  F-REX  was  used  (Andersson  2009; 
Pilemalm,  Andersson  &  Hallberg,  2008).  This  approach  allowed  subject  matter  experts  to 
trace  back  and  analyze  events  of  interest  that  had  happened  earlier  during  the  simulation 
runs. 
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Results 


In  this  study,  emphasis  lies  on  the  prerequisites  for  C2.  Factor  analysis,  multidimensional 
scaling  (MDS)  (Davison  &  Sireci,  2000;  Young,  1985)  and  LISREL  (Joreskog  & 

Sorbom,  1993;  Diamantopoulus  &  Siguaw,  2000)  analyses  of  participants’  and  observers’ 
ratings  of  these  prerequisites  were  perfonned  in  order  to  identify  how  the  prerequisites 
clustered  and  to  analyze  relations  between  the  different  prerequisites.  In  parallel  to  this, 
the  experiment  results  included  for  example  performance  assessment  using  an  objective 
performance  measure,  qualitative  analysis  of  situational  awareness  and  reliability 
analysis.  The  qualitative  situational  awareness  analysis  and  the  performance  analysis  are 
presented  briefly  in  this  paper  since  they  are  of  interest  in  the  methodological  discussion 
and  provide  a  context  to  the  C2  ratings. 

As  described  in  the  methods  section,  the  survey  with  the  prerequisites  for  C2  was 
completed  by  participants  and  observers  during  three  to  four  times  each  day  and  included 
13  items  (one  for  each  prerequisite,  however  trust  was  three-fold  and  a  total  value  for  C2 
ability  was  rated).  A  multiple  analysis  of  regression  was  perfonned  based  on  these 
ratings,  in  order  to  analyze  to  what  extent  the  ten  prerequisites  for  C2  explained  the  rating 
of  C2  ability  as  a  whole.  The  prerequisites  (12  items)  were  used  as  predictors  and  the 
total  C2  ability  rating  was  used  as  criteria.  The  analysis  resulted  in  the  multiple 
conelation  coefficient  of  R  =  .83,  p  <0.01 ,  R2  =  .70.  That  is,  the  prerequisites  explained 
70%  of  the  variance  of  the  rating  of  C2  ability  as  a  whole.  It  was  found  that  Operational 
picture  (40%),  Trust  for  technology  (37%),  Situational  Awareness  (42%)  and  Decision 
Making  (31%)  had  the  largest  amount  of  common  variance  with  the  C2  ability  as  a 
whole.  However,  all  correlations  were  significant,  that  is,  based  on  this  analysis,  all 
prerequisites  seem  relevant  for  how  the  observers  and  staff  members  rate  C2  ability  as  a 
whole.  The  MDS  analysis  showed  the  graphical  representation  of  the  ratings  of  single 
prerequisites  and  C2  ability  as  a  whole  (Figure  3).  The  shape  of  this  graphical 
representation,  in  which  C2  ability  is  placed  in  the  centre  and  all  other  prerequisites  are 
placed  around  this  item,  has  been  obtained  in  earlier  experiments  with  the  demonstrator 
(Hammervik,  2007).  As  seen  in  the  Figure  3,  the  prerequisites  are  all  separate,  that  is, 
from  this  analysis  no  prerequisites  can  be  interpreted  as  identical  items.  Factor  analysis 
was  used  to  identify  the  clusters  which  are  circled  in  Figure  3. 

The  factor  analysis  (extraction  with  Principal  Axis  Factoring,  Oblimin  rotation  with 
Kaiser  Nonnalization)  included  all  prerequisites,  but  not  the  total  value  of  C2  ability 
(KMO  =  .84;  Bartlett’s  test  of  sphericity,  =  696.16,  df  =  66,  p  <  .001).  It  resulted  in 
three  factors  similar  to  Figure  3.  The  Mission  intent  prerequisite,  however,  was  included 
in  the  factor  with  the  prerequisites  Trust  A  (trust  for  staff  assistant),  Trust  C  (trust  for 
other  staff  members)  and  Teamwork.  Since  Mission  intent  had  the  lowest  value  in  this 
factor,  a  separate  factor  analysis  was  conducted  on  this  factor  (KMO  =  .75;  Bartlett’s  test 
of  sphericity,  y  =  248.81,  df  =  6,p  <  .001).  According  to  this  analysis,  Mission  intent 
could  be  considered  as  a  separate  factor,  which  is  illustrated  by  the  circles  in  Figure  3. 
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Derived  Stimulus  Configuration 


Dimension  1 

Figure  3.  MDS  analysis  of  the  prerequisites  for  C2  (Trust  A=  Staff  assistant,  Trust  B  =  technology, 
Trust  C  =  staff  members).  According  to  the  factor  analysis  on  which  the  LISREL  is  based,  the  circled 
terms  are  related. 

Reliability  analysis  of  the  clusters  in  Figure  3  showed  high  reliability  of  the  situation 
factor,  a  =  .82  (Chronbach’s  alpha),  and  the  team  factor,  a  =  .86,  and  acceptable 
reliability  of  the  decision  factor,  a  =  .72  (for  factor  naming  see  Figure  4) 

The  pattern  of  the  MDS  analysis  was  confirmed  and  further  explored  in  a  causal  analysis 
with  linear  structural  equation  modelling  using  the  LISREL  software  (Joreskog  & 
Sorbom,  1993).  The  LISREL  model  produced  is  shown  in  Figure  4.  During  analysis,  four 
latent  variables  were  created  (in  Figure  4  shown  as  ovals),  explaining  the  variance 
between  the  manifest  variables  (shown  as  squares  in  the  figure).  RMSEA  (Root  Mean 
Square  Error  of  Approximation)  =  0.063;  CFI  (Comparative  Fit-Index)  =  0.96,  showing  a 
good  fit  of  the  model. 
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Figure  4.  LISREL  model  describing  the  relations  between  the  eleven  manifest  variables  (squares),  the 
four  latent  variables  (ovals)  and  the  factor  loads. 


As  shown  in  the  LISREL  model  in  Figure  4,  Mission  intent  affects  both  the  Team  factor 
(.61)  and  the  situation  factor  (.30),  with  larger  impact  on  the  Team  factor.  The  Situation 
factor  affects  both  the  team  factor  (.27)  and  the  Decision  factor  (.30).  The  Team  factor 
has  larger  impact  on  the  Decision  factor  (.61)  than  the  Situation  factor  has  (.30).  That  is, 
the  most  important  path  in  the  model  is  Mission  intent  — »  Team  — »  Decision. 

The  latent  variables  are  principally  the  same  as  the  clusters/factors  in  the  MDS  and  the 
factor  analyses,  except  for  Feedback  which  was  excluded  in  the  LISREL  model,  since  it 
had  significant  loadings  on  the  three  latent  variables  -  team,  situation  and  decision. 

Situation  awareness  was  qualitatively  analysed  in  that  comparisons  were  made  between 
system  logs  and  the  participants’  estimations  of  when  they  were  exposed  to  electronic 
attacks  (radio  or  radar  jamming)  or  computer  network  operations.  Results  of  the  actual 
EW  attacks  and  CNO  were  found  in  the  system  logs,  while  the  results  of  whether  the  staff 
considered  themselves  exposed  to  such  activities  were  generated  from  the  NBOT  tool  in 
which  these  data  were  reported  by  the  staff  with  4-minutes  interval  or  as  soon  as  they 
discovered  that  they  were  exposed  to  CNO  or  EW  attack.  Exposure  to  CNO  or  EW  could 
be  detected  in  different  ways  by  the  staff,  such  as  warnings  by  IDS,  reports  or  lack  of 
reports  from  subordinate  units,  signs  of  information  leakage  or  infonnation  distortion. 

For  each  experiment  and  phase,  a  graph  was  produced,  showing  experienced  and  actual 
exposure  to  hostile  EW  and  CNO.  An  example  from  the  second  phase  of  the  first 
experiment  is  shown  in  Figure  5.  Interesting  events  could  then  be  noted,  for  instance 
when  staff  had  noted  jamming  although  logs  showed  that  there  was  no  jamming,  and 
when  they  did  not  recognize  that  they  were  exposed  to  an  attack.  These  situations  were 
analyzed  using  the  F-REX  tool.  These  specific  events  are  marked  with  a  red  circle  in 
Figure  5  and  explained  in  Table  1.  Analysis  of  actual  and  experienced  jamming/CNO 
show  that  SA  was  quite  high  among  participants,  since  most  actual  jamming  and  CNO 
was  detected  by  the  participants.  However,  from  the  graph  it  seems  as  the  staff,  although 
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that  they  detected  that  they  were  exposed  to  an  attack,  they  did  not  immediately  realise 
which  type  of  attack,  and  therefore  they  noted  that  they  were  exposed  to  both 
communication  jamming  and  CNO.  Replay  of  the  communication  within  the  staff 
revealed  that  after  a  while,  they  usually  detected  the  correct  type  of  attack.  Thus,  the 
graph  may  be  used  as  a  tool  to  be  able  to  give  a  first  general  overview  on  the  staffs 
situational  awareness,  but  for  a  detailed  and  fair  analysis,  it  is  necessary  to  replay  data 
sources  for  the  critical  events  identified.  That  is  what  has  been  done  in  order  to  generate 
table  1. 


Evacuation  phase 

(Experiment  1  Day  2) 


Actual  and  Experienced  Jamming/CNO 

|  Actual  jamming/CNO  |  Experienced  jamming/CNO  and  impact  on  C2  ability 


High  impact 
Low  impact 
No  jamming 
Jamming 
No  Jamming 


© 


0 


0 


High  impact 
Low  impact 
No  jamming 
Jamming 
No  Jamming 


High  impact 
Low  impact 
No  CNO 
CNO 
No  CNO 


©  Note  number 


© 


i© 


© 


© 


i© 


Time  of  day  [hh:mm] 


Figure  5.  Example  of  actual  and  experienced  exposure  to  communication  jamming,  radar  jamming 
and  CNO  (experiment  1,  evacuation  phase)  against  the  staff.  When  the  staff  experienced  exposure, 
they  also  rated  whether  it  had  a  high  or  low  impact  on  their  C2  ability. 


Table  1.  F-REX  analysis  of  specific  events  found  during  comparison  of  actual  and  experienced 
_ _  exposure  to  EW  and  CNO  (experiment  1,  evacuation  phase). _ 


Note 

Time 

Comment 

1 

09:38 

Intrusion  in  mail  server.  Discovered  by  the  staff  at  10:51,  see  note  2. 

2 

10:51 

Staff  discovers  that  information  about  IP  addresses  and  mail  conversation  within  the 
force  is  published  on  Internet. 

3 

11:03 

Staff  interprets  the  information  leakage  in  note  2  as  communication  jamming. 

4 

11:08 

The  staff  receives  an  e-mail  with  HIC  as  sender,  however  this  is  a  false  e-mail,  which 
the  staff  immediately  detects.  Staff  informs  HIC  by  using  an  alternative  line  of 
communication  (radio). 

5 

11:35 

Communication  jamming  of  GPS,  staff  detects  that  positions  of  own  units  are  not 
updated. 

6 

13:18 

The  dailv  newspaper  www.aftonblaskan.sl  is  hacked  according  to  the  loa.  However,  F- 
REX  analysis  shows  that  this  intrusion  is  not  indicated  in  the  interfaces  in  a  way  that  the 
staff  can  see  it. 

7 

13:56 

The  note  on  CNO  is  probably  a  false  note  by  the  staff  member,  F-REX  analyses  show 
that  hostile  CNO  is  not  indicated  in  simulated  systems,  and  staff  does  not  discuss  being 
attacked  at  the  current  event. 
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Performance  was  assessed  for  both  phases  of  each  experiment  using  the  objective 
performance  measure.  Results  show  that  the  staff  in  Experiment  1  performed  60%  of 
maximum  score  (60%  day  1,61%  day  2)  and  staff  in  Experiment  2  performed  69%  (69% 
day  1,  70%  day  2).  One  reason  to  differences  in  perfonnance  may  be  that  the  staff 
members  in  experiment  2  were  more  familiar  working  with  each  other.  Differences  in 
knowledge  and  experience  is  a  possible  explanation,  however  not  plausible  since  the 
differences  in  knowledge  and  experience  rather  pointed  to  that  staff  members  in 
Experiment  1  had  advantages  in  tenns  of  competence  and  experience.  Qualitatively,  the 
performance  is  assessed  to  be  good,  since  both  staffs  managed  to  solve  most  of  the  issues 
appearing  during  the  simulation. 


Discussion 

From  MDS  and  LISREL  analyses  it  can  be  concluded  that  the  prerequisites  seem  to 
group  into  two  main  factors  -  the  Situation  and  the  Team  factor.  The  Mission  intent 
affects  both  of  these  factors  and  these  in  turn  affect  the  Decision.  The  prerequisites  that 
form  the  team  factor  seem  to  affect  the  decision  more  than  prerequisites  related  to  the 
Situation  factor.  The  feedback  prerequisite  needs  more  investigation,  it  is  not  clear 
whether  it  should  be  reformulated,  removed  or  incorporated  in  another  prerequisite.  Both 
participants  and  observers  thought  that  this  prerequisite  was  particularly  difficult  to 
estimate,  and  perhaps  it  was  not  described  to  them  in  a  clear  way.  These  results  are  based 
on  observers’  and  participants’  ratings  during  the  experiment,  that  is,  the  actual  quality  of 
the  decisions  was  not  known  but  it  was  rather  the  flexibility  and  the  timeliness  of 
decisions  that  were  assessed.  Since  these  results  are  based  on  ratings,  the  next  step  would 
be  to  analyze  prerequisites  from  objective  measures.  However  one  must  note  that 
participants’  ratings  of  some  of  the  prerequisites  may  be  more  reliable  than  observers’  or 
other  types  of  “objective”  measures  (such  as  trust  and  knowledge),  and  some 
prerequisites  may  not  be  interesting  or  possible  to  assess  using  any  kind  of  objective 
measure. 

The  performance  measure  enabled  quantitative  comparisons  of  the  teams.  The  F-REX 
synchronized  replay  of  all  recorded  sources  was  what  enabled  the  actual  performance 
analysis  (Pilemalm,  Andersson  &  Hallberg,  2008).  Still,  scoring  was  rather  time 
consuming  since  most  work  had  to  be  done  manually.  A  remaining  issue  is  then  how  to 
automate  an  objective  and  reliable  performance  measure  in  a  dynamic,  event-driven 
scenario  such  as  this?  The  scenario  was  realistic,  complex  and  provided  learning 
opportunities  since  game  control  and  HIC  responded  in  accordance  to  the  actions  of  the 
staff,  but  resource  consuming  in  terms  of  analysis.  Furthermore,  a  difficult  issue  when 
developing  a  performance  measure  is  how  to  credit  innovative  solutions  and  creativity 
that  the  developers  of  the  perfonnance  scale  could  not  foresee?  A  demonstrator 
environment  provides  the  ability  to  compare  subjective  and  objective  measures,  which 
during  these  experiments  was  particularly  useful  when  assessing  situational  awareness 
(based  on  experienced/actual  jamming/CNO).  Another  advantage  of  using  a  simulated 
environment  is  that  the  situation  can  be  manipulated  in  a  controlled  way.  In  this  case,  C2 
warfare  provided  an  opportunity  to  manipulate  the  participants’  C2  ability.  Due  to  the 
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fact  that  the  participants  were  subject  matter  experts;  they  were  still  able  to  recognize 
situations  and  use  tools  and  tactics  that  they  would  use  in  a  real  setting. 


Conclusions 

What  is  C2  ability?  Results  of  these  experiments  point  to  that  the  prerequisites  which 
form  the  hypothesis  used  during  these  experiments  are  central  aspects  that  need  to  be 
considered  when  trying  to  increase  or  protect  staffs’  C2  ability.  The  C2WD  provided  an 
environment  in  which  C2  could  be  studied,  in  a  more  controlled  way  than  in  a  real 
setting,  but  still  a  complex  and  realistic  environment.  Based  on  the  results,  it  can  be 
concluded  that  team  aspects  are  central  regarding  the  timeliness  and  flexibility  in 
decisions,  perhaps  more  central  than  that  the  situation  is  interpreted  correctly. 
Development  of  an  objective  performance  measure  aided  in  comparing  the  teams,  and  a 
remaining  challenge  is  to  find  a  solution  of  how  to  automate  such  a  measure,  in  order  to 
be  able  to  give  immediate  feedback  to  the  teams  and  make  analyses  less  time  consuming. 
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•  C2  Warfare  Demonstrator  technology  and  methods 
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Hypothesis 


C2W  Demonstrator 


•  Electronic  Warfare  and  Computer  Network  Operations 

•  Detection,  countermeasures,  prevention 

•  Radar,  signal  intelligence,  Infra  red  search  and  track  (IRST),  jammers 
(radio,  radar,  mobile  telephones)... 

•  Network  scan,  Denial  of  Service,  spyware,  Intrusion  detection  system 


(IDS)... 


•  Communication 

•  Radio,  mobile  telephone,  satellite,  e-mail 

•  Three-dimensional  terrain  model 

•  Web  pages 

•  Views  -  Network,  sensors,  e-mail  inbox,  internet 


Scenario 


•  International  force  deployed  in  a  fictitious 
country 

•  Staff  responsible  for  EW  and  CNO  resources 

•  Irregular  and  regular  forces,  criminal  actors 

•  Task:  Support  the  evacuation  of  NGO:s 

•  Intelligence  phase,  evacuation  phase 
N  •  Keys  to  success 

•  Social  network  analysis 
H  •  Locate  spoilers  to  peace  process 

•  Neutralise  spoilers 


Method 


Two  experiments  conducted 
Three  days 

•  Training 

•  Intelligence  phase 

•  Evacuation  phase 
Participants:  Staff  of  3-4  persons 

•  CNO  and  EW  experts 
Game  control 

•  Higher  chief  in 
command 

•  Staff  assistants 

•  Opposing  actors 
Observers 


Data  collection 


System  logs 
Video  and  audio 
Surveys 
Observers 

•  C2  aspects 

•  Communication 
Performance  measure 
Experienced  jamming/CNO 
Estimations  of  prerequisites  for  C2 


Results 

Analysis  of  Situational  Awareness  and  Performance 
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Results 

Performance 


•  Experiment  1 

•  Intelligence  phase  60% 

•  Evacuation  phase  60% 

•  Experiment  2 

•  Intelligence  phase  70% 

•  Evacuation  phase  70% 

•  Overall  good  performance,  most  issues  solved 

•  Staff  in  experiment  2  more  familiar  with  each  other? 


Dimension  2 


Results 

Multidimensional  scaling  of  prerequisites 


Derived  Stimulus  Configuration 


Results 

LISREL  model  of  prerequisites 
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Information 
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picture 

experience 

Technology 

flow 
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Conclusions 


•  Actual/experienced  jamming 

•  Good  overview  of  Team  SA 

•  1 .  Automatically  generated  statistics,  2.  FREX  analysis 

•  Performance  measure 

•  Aided  in  comparing  the  teams 

•  Resource  consuming  -  automate? 

•  Prerequisites 

•  Separate,  but  can  be  clustered 

•  Team  issues  higher  impact  on  decision  than  SA 
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